Providing contextual personal information by a mixed reality device

ABSTRACT

The technology provides contextual personal information by a mixed reality display device system being worn by a user. A user inputs person selection criteria, and the display system sends a request for data identifying at least one person in a location of the user who satisfy the person selection criteria to a cloud based application with access to user profile data for multiple users. Upon receiving data identifying the at least one person, the display system outputs data identifying the person if he or she is within the field of view. An identifier and a position indicator of the person in the location is output if not. Directional sensors on the display device may also be used for determining a position of the person. Cloud based executing software can identify and track the positions of people based on image and non-image data from display devices in the location.

BACKGROUND

Social networking and their varieties such as professional networkingsites like Linked-In® allow people to get a sense of a person'sbackground and connections. However, face to face connections often leadto more meaningful interactions and make impressions one does not getfrom reading about someone online or on paper. Then again, a person isoften unaware of relevant interests and information shared with anotherperson during a face to face conversation, or even that another personwith shared interests is in the same room.

SUMMARY

Mixed reality is a technology that allows virtual imagery to be mixedwith a real world view. A see-through, head mounted, mixed realitydisplay device system, also referred to as a see-through HMD system, maybe worn by a user to view real objects mixed with image data of one ormore virtual objects displayed in a field of view of the display device.Often, the mixed reality device system displays three-dimensional (3D)image data, for example a 3D hologram, in addition to text and videodata. The technology uses a mixed reality device system to display imagedata, output audio data or both which identifies one or more people inits field of view or in a same location as the user who satisfy theuser's person selection criteria. An example of person selectioncriteria is a job position indicating the person is a hiring decisionmaker for a programmer position with an experience level for a certainsoftware language. Another example of person selection criteria is allthe people present in a defined location with a single marital status.

The technology provides an embodiment of one or more processor readablestorage devices having instructions encoded thereon for causing one ormore software controlled processors to execute a method for providingcontextual personal information by a mixed reality display devicesystem. The method comprises receiving and storing person selectioncriteria associated with a user wearing the mixed reality display devicesystem. A request is sent including a location of the user and theperson selection criteria to a personal information service engine,which executes on a remote computer system, for a personalidentification data set for each person sharing the location andsatisfying the person selection criteria. At least one personalidentification data set is received from the personal identificationservice engine for a person sharing the location, and a determination ismade as to whether the person associated with the at least one personalidentification data set is in the field of view of the mixed realitydisplay device system. Responsive to the person associated with the atleast one personal identification data set being in the field of view,data is output by the display device system which identifies the personin the field of view.

The technology provides an embodiment of a method for providingcontextual personal information to a user wearing a head mounted, mixedreality display device system. The method comprises receiving a requestindicating a location of a user head mounted, mixed reality, displaydevice system and person selection criteria for at least one personalidentification data set for each person sharing the location andsatisfying the person selection criteria. A determination is made as towhether there is a person sharing the location and satisfying the personselection criteria based on accessible user profile data. Responsive tothere being a person sharing the location and satisfying the personselection criteria, generating at least one personal identification dataset is generated for the person, and it is determined whether the personassociated with the at least one personal identification data set iscurrently within a field of view of the user head mounted, mixedreality, display device system.

Responsive to the person not being currently within the field of view ofthe user display device system, a position of the person within thelocation is determined, and sent in the at least one personalidentification data set. Responsive to the person being currently withinthe field of view of the user display device system, and the at leastone personal identification data set including a personal identifier forthe person in the field of view is sent to the user display devicesystem.

The technology provides an embodiment of a system for a see-through,head mounted, mixed reality display device system for providingcontextual personal information. The system comprises a see-through,mixed reality, display positioned by a head mounted support structure.An example of a support structure is a frame. At least one front facingcamera is positioned on the support structure for capturing image dataof a field of view of the see-through display. One or more directionalsensors are attached to the support structure, and each has a sensorposition with reference to a body part of the user. Each of the one ormore directional sensors transmits an identity data set including thesensor position. One or more software controlled processors arecommunicatively coupled to the at least one front facing camera forreceiving the image data of the field of view. The one or moreprocessors also communicate with a remote computer system executing apersonal information service engine. The one or more processors send arequest with person selection criteria and a location of a user wearingthe head mounted support structure, and receive a personalidentification data set of a person sharing the location and satisfyingthe person selection criteria.

At least one image generation unit communicatively coupled to the one ormore software controlled processors and optically coupled to thesee-through, mixed reality display tracks virtual data to the personsharing the location and satisfying the person selection criteria in afield of view of the see-through display.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram depicting example components of oneembodiment of a see-through, mixed reality display device system.

FIG. 1B is a block diagram depicting example components of anotherembodiment of a see-through, mixed reality display device system.

FIG. 1C is a block diagram depicting example components of anotherembodiment of a see-through, mixed reality display device system using amobile device as a processing unit.

FIG. 2A is a side view of an eyeglass temple of a frame in an embodimentof the see-through, mixed reality display device embodied as eyeglassesproviding support for hardware and software components.

FIG. 2B is a top view of an embodiment of a display optical system of asee-through, head mounted, mixed reality device.

FIG. 3 is a block diagram of a system from a software perspective forproviding contextual personal information via a see-through, mixedreality display device system.

FIG. 4 is a flowchart of an embodiment of a method of operation of amixed reality display device system which provides contextual personalinformation.

FIG. 5 is a flowchart of an embodiment of a method for providingcontextual personal information to a mixed reality display devicesystem.

FIG. 6 is a flowchart of an embodiment of a method for providingcontextual personal information by a head mounted, mixed reality displaydevice system.

FIG. 7A is a flowchart of an embodiment of a process for determiningwhether a person associated with a respective personal identificationdata set is currently within a field of view of the user's mixed realitydisplay device system.

FIG. 7B is a flowchart of an embodiment of a process for determining aposition of a person associated with a respective personalidentification data set and not identified within the user field of viewbut who is associated with a head mounted, mixed reality display devicesystem in the location.

FIG. 7C is a flowchart of an embodiment of a process for determining aposition of a person within the location and associated with arespective personal identification data set responsive to the person notbeing within the user field of view but who has been identified in fieldof view data of another head mounted, mixed reality, display devicesystem.

FIG. 7D is a flowchart of an embodiment of a process for determining aposition of a person within the location and associated with arespective personal identification data set responsive to the person notbeing within the user field of view but for whom non-image location dataindicates a position.

FIG. 8A is a flowchart of an embodiment of a process for trackingpersonal interactions with one or more people in a location.

FIG. 8B is a flowchart of an embodiment of a process for determining apersonal interaction status for a person.

FIG. 9 illustrates an example of providing contextual personalinformation to a user wearing a head mounted, mixed reality displaydevice system.

FIG. 10 is a block diagram of one embodiment of a computing system thatcan be used to implement a network accessible computing system.

FIG. 11 is a block diagram of an exemplary mobile device which mayoperate in embodiments of the technology.

DETAILED DESCRIPTION

The technology provides various embodiments for providing contextualpersonal information by a head mounted, mixed reality display devicesystem. A user generates selection criteria either via user input to hisor her head mounted mixed reality device system or another usercomputing device. The user HMD outputs virtual data identifying one ormore individuals or people who satisfy the criteria. For example,individuals attending an event grant permission to personal informationdata items either through categories or by granting access to userprofile data maintained by one or more applications. The mixed realitydevice outputs data typically by displaying image data registered to theperson when in the field of view of the user device. Audio output mayalso be used. A personal identifier, like a name, and a relativeposition to the user may be displayed for a person not in the user'sdevice field of view. The phrase “user field of view” refers to thefield of view of the display device of the user which is captured byfront facing cameras on the display device as their field of viewoverlaps what the user sees with his or her own eyes. In someembodiments, additional cameras may be attached to the display device toprovide a wider field of view, for example, a 360 degree field of view.

The technology not only identifies people within a user field of view ora wider device field of view who satisfy the criteria, but alsoidentifies a position of a person satisfying the criteria who is outsidethe field of view but within the location of the user. An image mappingof a location may be available from a computer system under the controlof the location. For example, a conference center may have one or morecameras, perhaps depth cameras, at different points in the locationwhich image map the location during a conference the user is attending.An image map of a location at a certain time may also be made by imagedata captured from different head mounted displays in the location whichis uploaded to a location modeling application. An example of such anapplication is Photosynth® which creates a three-dimensional model of alocation. Through display device systems which cooperate to share theirimage data of their fields of view as well as data from non-imagedirectional sensors, a remote personal information service engine candetermine a position of a person in the location, in some instances evenif the person is not wearing a head mounted, mixed reality displaydevice system. A personal information service engine shares data ofdifferent users which subscribe to the service in accordance with theirpermissions. Some examples of personal information service engines aresocial and professional networking sites like Facebook® and LinkedIn®.

In other examples, a local version of the personal information servicemay be executing on a server in a location, and user can register withthe local personal information service engine when in the location. Auser may grant permission for use of user profile data only in thelocation, and only for a specific event. The server can search fasterfor people satisfying person selection criteria in these examples as itonly searches those registered for the event.

Additionally, person to person interactions with the identified peopleor other people can be tracked as well. Interactions can be classifiedbased on physical actions of the person and the user. Physical actionsare natural movements of one or more body parts of a user or otherperson. Physical actions such as gaze, gaze duration and blinking can bedetected from eye data generated by one or more eye tracking assemblies.Voice data can be detected from a microphone and sound recognitionsoftware deciphers the meaning of words spoken or sounds. A gesture isanother example of a physical action, and can be detected by at leastone field of view camera of the user HMD provided the gesture isperformed within the field of view of the user HMD.

FIG. 1A is a block diagram depicting example components of an embodimentof a see-through, augmented or mixed reality display device system.System 8 includes a see-through display device as a head mounted displaydevice 2 in communication with a processing unit 4 via a wire 6 in thisexample or wirelessly in other examples. In this embodiment, headmounted, display device 2 is in the shape of eyeglasses in a frame 115,with a display optical system 14 for each eye in which image data isprojected into a user's eye to generate a display of the image datawhile a user also sees through the display optical systems 14 for anactual direct view of the real world.

The use of the term “actual direct view” refers to the ability to seereal world objects directly with the human eye, rather than seeingcreated image representations of the objects. For example, lookingthrough glass at a room allows a user to have an actual direct view ofthe room, while viewing a video of a room on a television is not anactual direct view of the room. Each display optical system 14 is alsoreferred to as a see-through display, and the two display opticalsystems 14 together may also be referred to as a see-through display.

Frame 115 provides a support structure for holding elements of thesystem in place as well as a conduit for electrical connections. In thisembodiment, frame 115 provides a convenient eyeglass frame as supportfor the elements of the system discussed further below. The frame 115includes a nose bridge portion 104 with a microphone 110 for recordingsounds and transmitting audio data to control circuitry 136 in thisembodiment. A temple or side arm 102 of the frame rests on each of auser's ears. In this example, the right temple 102 includes controlcircuitry 136 for the display device 2.

As illustrated in FIGS. 2A and 2B, an image generation unit 120 isincluded on each temple 102 in this embodiment as well. Also, not shownin this view, but illustrated in FIGS. 2A and 2B are front facingcameras 113 for recording digital images and videos and transmitting thevisual recordings to the control circuitry 136 which may in turn sendthe captured image data to the processing unit 4 which may also send thedata to one or more computer systems 12 over a network 50.

The processing unit 4 may take various embodiments. In some embodiments,processing unit 4 is a separate unit which may be worn on the user'sbody, e.g. a wrist, or be a separate device like the illustrated mobiledevice 4 as illustrated in FIG. 1C. The processing unit 4 maycommunicate wired or wirelessly (e.g., WiFi, Bluetooth, infrared, RFIDtransmission, wireless Universal Serial Bus (WUSB), cellulartelecommunication, 3G, 4G or other wireless communication means) over acommunication network 50 to one or more computing systems 12 whetherlocated nearby or at a remote location. In other embodiments, thefunctionality of the processing unit 4 may be integrated in software andhardware components of the display device 2 as in FIG. 1B.

A remote, network accessible computer system 12 may be leveraged forprocessing power and remote data access. An example of hardwarecomponents of a computing system 12 is shown in FIG. 10. Computingsystem 12 may be implemented using one or more computer systems. Inthese examples, the computing system 12 is also, optionally,communicatively coupled to one or more position sensors 52 within alocation. Besides examples discussed below such as infrared, WUSB, RFIDand directional antennas, a depth camera in a location may also becoupled for receiving three-dimensional (3D) image data of the locationfrom which real objects and their positions in 3D can be identified.

In other examples, the one or more position sensors 52 within thelocation may include non-image data directional sensors such as a WiFinetwork access point with a directional antenna for picking up wirelesssignals from see-through, mixed reality display devices. The location ofthe WiFi access point within a building or other geographic location canbe stored by the computer system 12. In some estimates, the range of aWiFi access point is about 20 to 30 meters indoors while 60 to 90 metersoutdoors depending on antenna strength. Directional position sensorswith smaller ranges can also be deployed in a location.

FIG. 1B is a block diagram depicting example components of anotherembodiment of a see-through, augmented or mixed reality display devicesystem 8 which may communicate over a communication network 50 withother devices. In this embodiment, the control circuitry 136 of thedisplay device 2 communicates wirelessly via a wireless transceiver (see137 in FIG. 2A) over a communication network 50 to one or more computersystems 12.

FIG. 1C is a block diagram of another embodiment of a see-through, mixedreality display device system 8 using a mobile device as a processingunit 4. Examples of hardware and software components of a mobile device4 such as may be embodied in a smart phone or tablet computing deviceare described in FIG. 17. A display 7 of the mobile device 4 may alsodisplay data, for example menus, for executing applications and be touchsensitive for accepting user input. Some other examples of mobiledevices 4 are a laptop or notebook computer, and a netbook computer.

FIG. 2A is a side view of an eyeglass temple 102 of the frame 115 in anembodiment of the see-through, mixed reality display device 2 embodiedas eyeglasses providing support for hardware and software components. Atthe front of frame 115 is physical environment facing video camera 113that can capture video and still images of the real world to map realobjects in the field of view of the see-through display, and hence, inthe field of view of the user. The cameras are also referred to as frontfacing cameras. Each front facing camera 113 is calibrated with respectto a reference point of its respective display optical system 14 suchthat the field of view of the display optical system 14 can bedetermined from the image data captured by the respective camera 113.One example of such a reference point is an optical axis (see 142 inFIG. 2B) of its respective display optical system 14. The image data istypically color image data.

In many embodiments, the two cameras 113 provide overlapping image datafrom which depth information for objects in the scene may be determinedbased on stereopsis. In some examples, the cameras may also be depthsensitive cameras which transmit and detect infrared light from whichdepth data may be determined The processing identifies and maps theuser's real world field of view. Some examples of depth sensingtechnologies that may be included on the head mounted display device 2without limitation, are SONAR, LIDAR, Structured Light, and/or Time ofFlight.

Control circuits 136 provide various electronics that support the othercomponents of head mounted display device 2. In this example, the righttemple 102 includes control circuitry 136 for the display device 2 whichincludes a processing unit 210, a memory 244 accessible to theprocessing unit 210 for storing processor readable instructions anddata, a wireless interface 137 communicatively coupled to the processingunit 210, and a power supply 239 providing power for the components ofthe control circuitry 136 and the other components of the display 2 likethe cameras 113, the microphone 110 and the sensor units discussedbelow. The processing unit 210 may comprise one or more processorsincluding a central processing unit (CPU) and a graphics processing unit(GPU).

In some examples, the wireless interface 137 includes a directionalantenna 145 which can act as a directional relative position sensor.When an identity data set, which may include, for example, a deviceidentifier, a time stamp and a personal identifier of a logged in user,from another display device system 8 or another computing system like amobile device is received, the processing unit 210 can determine arelative position of the transmitting device and the person identifiedin the identity data set because of the directional antenna 145. Thetime stamp is a basis for identifying a current position of the personbeing sought.

Inside, or mounted to temple 102, are ear phones 130, inertial sensors132, one or more location sensors 144 and directional relative positionsensors 145 at different positions on the temple 102. An example of alocation sensor is a Global Positioning System (GPS) device which canprovide location information independent of using the display system 8as a reference. GPS and wireless transceivers like transceiver 137 canprovide detection for ranges of about 10 or more meters. However, in acrowded room, more precision may be desirable.

In many examples, a directional position sensor is a wireless device,like a receiver or transceiver, which detects a wireless signal fromanother mixed reality device or computing device and typically alsotransmits a wireless signal. Some examples of directional positionsensors 145 are an infrared (IR) device, a WUSB transceiver, a Bluetoothdevice or a radio frequency device for processing RFID data. In theembodiment of FIG. 2A, the placement of the detection areas of thesensors 145 with respect to the temple 102 indicates from whichdirection with respect to the temple a sensor has received a signal.Assuming a user is wearing the display device 2, the direction isassumed to be with respect to the user's head. For example, a signalreceived at the sensor in the middle of the right temple is determinedto be from a device on the right side of the user's head. In otherembodiments, a body part reference other than the head may be used. Thesignal range set for the sensor provides an approximate distanceboundary within which is positioned a detected display or mobile devicefrom the user's display device 2.

For example, a wireless universal serial bus (WUSB) device may form awireless connection with a mixed reality display device system 8 withinapproximately three to ten meters. Bluetooth may also be used and formwireless connections with other Bluetooth devices at a distance of about1 meter if desired or much higher such as 100 meters depending on theversion of Bluetooth used and signal power used. For distances of lessthan a meter such as for a space about a person or a particular object,infrared signals may be used to exchange data for example using InfraredData Association (IrDA) communication protocols. Depending on theimplementing technology, RFID technology can provide different detectiondistances as well. For example, some RFID technology can detect anobject from less than a few centimeters while other technology candetect an object within 60 feet. As described further in the figuresbelow, non-image data from these directional sensors can be processedmuch quicker than image data and can reduce time for finding a positionof a person.

Optional electrical impulse sensor 128 detects commands via eyemovements. In one embodiment, inertial sensors 132 include a three axismagnetometer 132A, three axis gyro 132B and three axis accelerometer132C. The inertial sensors are for sensing position, orientation, andsudden accelerations of head mounted display device 2. From thesemovements, head position with respect to the ground may also bedetermined In this embodiment, each of the devices using an analogsignal in its operation like the sensor devices 144, 145, 128, 130, and132 as well as the microphone 110 and an IR illuminator 134A discussedbelow, include control circuitry which interfaces with the digitalprocessing unit 210 and memory 244 and which produces and convertsanalog signals for its respective device.

Mounted to or inside temple 102 is an image source or image generationunit 120 which produces visible light representing images. In oneembodiment, the image source includes microdisplay 120 for projectingimages of one or more virtual objects and coupling optics lens system122 for directing images from microdisplay 120 to reflecting surface orelement 124. The microdisplay 120 may be implemented in varioustechnologies including transmissive projection technology, micro organiclight emitting diode (OLED) technology, or a reflective technology likedigital light processing (DLP), liquid crystal on silicon (LCOS) andMirasol® display technology from Qualcomm, Inc. The reflecting surface124 directs the light from the microdisplay 120 into a lightguideoptical element 112, which directs the light representing the image intothe user's eye. Image data of a virtual object may be registered to areal object meaning the virtual object tracks its position to a positionof the real object seen through the see-through display device 2 whenthe real object is in the field of view of the see-through displays 14.

FIG. 2B is a top view of an embodiment of one side of a see-through,head mounted, mixed reality display device including a display opticalsystem 14. A portion of the frame 115 of the head mounted display device2 will surround a display optical system 14 for providing support andmaking electrical connections. In order to show the components of thedisplay optical system 14, in this case 14 r for the right eye system,in the head mounted display device 2, a portion of the frame 115surrounding the display optical system is not depicted.

In the illustrated embodiment, the display optical system 14 is anintegrated eye tracking and display system. The system includes a lightguide optical element 112, opacity filter 114, and optional see-throughlens 116 and see-through lens 118.

Light guide optical element 112 transmits light from microdisplay 120 tothe eye 140 of the user wearing head mounted, display device 2. Lightguide optical element 112 also allows light from in front of the headmounted, display device 2 to be transmitted through light guide opticalelement 112 to eye 140, as depicted by arrow 142 representing an opticalaxis of the display optical system 14 r, thereby allowing the user tohave an actual direct view of the space in front of head mounted,display device 2 in addition to receiving a virtual image frommicrodisplay 120. Light from microdisplay 120 passes through lens orlens system 122 and becomes incident on reflecting surface 124. Thereflecting surface 124 reflects the incident light from the microdisplay120 such that light is trapped inside a waveguide, a planar waveguide inthis embodiment. A representative reflecting element 126 represents theone or more optical elements like mirrors, gratings, and other opticalelements which direct visible light representing an image from theplanar waveguide towards the user eye 140.

Opacity filter 114, which is aligned with light guide optical element112, selectively blocks natural light from passing through light guideoptical element 112 for enhancing contrast of virtual imagery. Moredetails of an opacity filter are provided in U.S. patent applicationSer. No. 12/887,426, “Opacity Filter For See-Through Mounted Display,”filed on Sep. 21, 2010, incorporated herein by reference in itsentirety.

The position of the user's eyes and image data of the eye in general maybe used for applications such as gaze detection, and blink commanddetection. The eye tracking system 134 comprises an eye trackingillumination source 134A, e.g. LED or laser, emitting about apredetermined IR wavelength and an eye tracking IR sensor 134B, e.g. IRcamera or IR position sensitive glint detector (PSD), positioned betweenlens 118 and temple 102 in this example.

In this embodiment, a wavelength selective filter 123 passes throughvisible spectrum light from the reflecting surface 124 and directs theinfrared wavelength illumination from the eye tracking illuminationsource 134A into the light guide optical element 112, e.g. a planarwaveguide, through wavelength selective filter 125 in a directionheading toward the nose bridge 104. Reflective element 126 in thisexample is also representative of one or more optical elements whichimplement bidirectional infrared filtering which directs IR illuminationtowards the eye 140, preferably centered about the optical axis 142 andreceives IR reflections from the user eye 140. The IR sensor 134B isalso optically coupled to the wavelength selective filter 125 whichdirects only infrared radiation from the waveguide including infraredreflections of the user eye 140, preferably including reflectionscaptured about the optical axis 142, out of the waveguide 112 to the IRsensor 134B.

In other embodiments, the eye tracking unit optics are not integratedwith the display optics. For more examples of eye tracking systems forHMD devices, see U.S. Pat. No. 7,401,920, entitled “Head Mounted EyeTracking and Display System”, issued Jul. 22, 2008 to Kranz et al., seeU.S. patent application Ser. No. 13/221,739, Lewis et al., entitled“Gaze Detection in a See-Through, Near-Eye, Mixed Reality Display,”filed Aug. 30, 2011, and see U.S. patent application Ser. No.13/245,700, Bohn, entitled “Integrated Eye Tracking and Display System,”filed Sep. 26, 2011, all of which are incorporated herein by reference.

Another embodiment for tracking the direction of the eyes is based oncharge tracking. This concept is based on the observation that a retinacarries a measurable positive charge and the cornea has a negativecharge. Sensors 128, in some embodiments, are mounted by the user's ears(near earphones 130) to detect the electrical potential while the eyesmove around and effectively read out what the eyes are doing in realtime. (See Control your mobile music with eyeball-activated earphones !,Feb. 19, 2010,http://www.wirefresh.com/control-your-mobile-music-with-eyeball-actvated-headphones,which is hereby incorporated by reference.) Eye blinks may be tracked ascommands Other embodiments for tracking eyes movements such as blinkswhich are based on pattern and motion recognition in image data from thesmall eye tracking camera 134B mounted on the inside of the glasses, canalso be used. The eye tracking camera 134B sends buffers of image datato the memory 244 under control of the control circuitry 136.

Again, FIGS. 2A and 2B only show half of the head mounted display device2. A full head mounted display device would include another set ofoptional see-through lenses 116 and 118, another opacity filter 114,another light guide optical element 112, another microdisplay 120,another lens system 122 front facing camera 113, eye tracking assembly134, earphones 130, directional sensors 145, and sensors 128 if present.Additional details of a head mounted display 2 are illustrated in U.S.patent application Ser. No. 12/905952 entitled Fusing Virtual ContentInto Real Content, Filed Oct. 15, 2010, fully incorporated herein byreference.

FIG. 3 illustrates a computing environment embodiment from a softwareperspective which may be implemented by the display device system 8, aremote computing system 12 in communication with the display devicesystem 8 or both. Network connectivity allows leveraging of availablecomputing resources. The computing environment 54 may be implementedusing one or more computer systems. As shown in the embodiment of FIG.3, the software components of a computing environment 54 include animage and audio processing engine 191 in communication with an operatingsystem 190. Image and audio processing engine 191 includes objectrecognition engine 192, gesture recognition engine 193, soundrecognition engine 194, virtual data engine 195, and eye trackingsoftware 196, all in communication with each other. Image and audioprocessing engine 191 processes video, image, and audio data receivedfrom a capture device such as the front facing cameras 113, othercameras on the display device (not shown) or optional cameras 52, e.g.depth cameras, in the location. To assist in the detection and/ortracking of objects, an object recognition engine 192 of the image andaudio processing engine 191 may access one or more databases ofstructure data 200 over one or more communication networks 50.

Virtual data engine 195 processes virtual objects and registers theposition and orientation of virtual objects in relation to one or morecoordinate systems. Additionally, the virtual data engine 195 performsthe translation, rotation, scaling and perspective operations usingstandard image processing methods to make the virtual object appearrealistic. A virtual object position may be registered or dependent on aposition of a corresponding real object. The virtual data engine 195determines the position of image data of a virtual object in displaycoordinates for each display optical system 14. The virtual data engine195 may also determine the position of virtual objects in various mapsof a real-world environment stored in a memory unit of the displaydevice system 8 or of the computing system 12. One map may be the fieldof view of the display device with respect to one or more referencepoints for approximating the locations of the user's eyes. For example,the optical axes of the see-through display optical systems 14 may beused as such reference points. In other examples, the real-worldenvironment map may be independent of the display device, e.g. a 3D mapor model of a location (e.g. store, coffee shop, museum).

One or more processors of the computing system 12, or the display devicesystem 8 or both also execute the object recognition engine 192 toidentify real objects in image data captured by the environment facingcameras 113 or other cameras 52 in the location. As in other imageprocessing applications, a person can be a type of object. For example,the object recognition engine 192 may implement pattern recognitionbased on structure data 200 to detect particular objects including ahuman. The object recognition engine 192 may also include facialrecognition software which is used to detect the face of a particularperson.

Structure data 200 may include structural information about targetsand/or objects to be tracked. For example, a skeletal model of a humanmay be stored to help recognize body parts. In another example,structure data 200 may include structural information regarding one ormore inanimate objects in order to help recognize the one or moreinanimate objects. The structure data 200 may store structuralinformation as image data or use image data as references for patternrecognition. The image data may also be used for facial recognition. Thestructure data 200 may include one or more image datastores includingimages of numbers, symbols (e.g. mathematical symbols), letters andcharacters from alphabets used by different languages.

In this embodiment, there is a cloud based location mapping engine 203which generates a model or map of a location in three dimensions basedon image data. It may store and access location images 170 usinglocation data, e.g. GPS data from GPS device 144, as an index. Anexample of such a mapping engine 203 is Photosynth®. Cameras 52 in thelocation or front facing cameras 113 and any other cameras on thesupport structures of one or more HMD display systems 8 in the locationcan upload their image data to the mapping engine 203. A mapping engine203, which can process depth data or create stereo images from the imagedata captured by cameras 113 and any other cameras on the supportstructure, can create a three dimensional model of a location whichtracks stationary and moving objects in the location based on localobject recognition by each display device system 8 of objects within itsrespective field of view. The mapping engine 203 can assign objectidentifiers to the objects for use by the mixed reality, display devicesystems 8 in the location.

Upon detection of one or more objects by the object recognition engine192, image and audio processing engine 191 may report to operatingsystem 190 an identification of each object detected and a correspondingposition and/or orientation which the operating system 190 passes alongto an application like the client personal information (PI) serviceapplication 204 _(1.)

The sound recognition engine 194 processes audio received via microphone110.

The front facing cameras 113 in conjunction with the gesture recognitionengine 193 implement a natural user interface (NUI) in embodiments ofthe display device system 8. Blink commands or gaze duration dataidentified by the eye tracking software 196 are also examples ofphysical action user input. Voice commands may also supplement otherrecognized physical actions such as gestures and eye gaze.

The gesture recognition engine 193 can identify actions performed by auser which may indicate a control or command to an executingapplication. For example, an exchange of hand waves indicate a personalinteraction of greeting to the personal information service engine orapplication 204 as discussed below. The action may be performed by abody part of a user, e.g. a hand or finger, but also an eye blinksequence of an eye can be gestures. In one embodiment, the gesturerecognition engine 193 includes a collection of gesture filters, eachcomprising information concerning a gesture that may be performed by atleast a part of a skeletal model. The gesture recognition engine 193compares a skeletal model and movements associated with it derived fromthe captured image data to the gesture filters in a gesture library toidentify when a user (as represented by the skeletal model) hasperformed one or more gestures. In some examples, a camera, inparticular a depth camera in the real environment separate from thedisplay device 2 in communication with the display device system 8 or acomputing system 12 may detect the gesture and forward a notification tothe system 8, 12. In other examples, the gesture may be performed inview of the cameras 113 by a body part such as the user's hand or one ormore fingers.

In some examples, matching of image data to image models of a user'shand or finger during gesture training sessions may be used rather thanskeletal tracking for recognizing gestures.

Also illustrated are a personal interaction rule set 178 which containsrules for use by the PI service application 204, 204 ₁ in categorizingan interaction as discussed further below.

More information about the detection and tracking of objects can befound in U.S. patent application Ser. No. 12/641,788, “Motion DetectionUsing Depth Images,” filed on Dec. 18, 2009; and U.S. patent applicationSer. No. 12/475,308, “Device for Identifying and Tracking MultipleHumans over Time,” both of which are incorporated herein by reference intheir entirety. More information about the gesture recognition engine193 can be found in U.S. patent application Ser. No. 12/422,661,“Gesture Recognizer System Architecture,” filed on Apr. 13, 2009,incorporated herein by reference in its entirety. More information aboutrecognizing gestures can be found in U.S. patent application Ser. No.12/391,150, “Standard Gestures,” filed on Feb. 23, 2009; and U.S. patentapplication Ser. No. 12/474,655, “Gesture Tool,” filed on May 29, 2009,both of which are incorporated by reference herein in their entirety.

The computing environment 54 also stores data in image and audio databuffer(s) 199. The buffers provide memory for receiving image datacaptured from the front facing cameras 113, image data from an eyetracking camera 134B of an eye tracking assembly 134 if used, buffersfor holding image data of virtual objects to be displayed by the imagegeneration units 120, and buffers for audio data such as voice commandsfrom the user via microphone 110 and instructions to be sent to the uservia earphones 130.

Device data 198 may include a unique identifier for the computer system8, a network address, e.g. an IP address, model number, configurationparameters such as devices installed, identification of the operationsystem, and what applications are available in the display device system8 and are executing in the display system 8 etc. Particularly for thesee-through, mixed reality display device system 8, the device data mayalso include data from sensors or determined from the sensors like thedirectional position sensors 145, the orientation sensors 132, thetemperature sensor 138, the microphone 110, the electrical impulsesensor 128 if present, and the one or more location sensors 144.

In this embodiment, the display device system 8 and other processorbased systems 161 used by the user execute a client side version of apersonal information service application 204 _(N) which communicatesover a communication network 50 with a personal information (PI) serviceengine or application 204 which is cloud based in this embodiment. Acloud based engine may include one or more software applicationinstantiations which execute on and store data by one or more networkedcomputer systems. The engine is not tied to a particular location. Someexamples of cloud based software are social networking sites likeFacebook® and LinkedIn® and web-based email sites like Yahoo! ® andHotmail®. A user may register an account with the personal informationservice engine 204 which grants the personal information servicepermission to monitor the user's executing applications and datagenerated and received by them which is stored in user profile data 197,and device data 198 for tracking the user's location and devicecapabilities. Based on the user profile data aggregated from the user'ssystems 8, 161, the data received and sent by the executing applicationson systems 8, 161 used by the user and stored in user profile data, andlocation and directional position sensor data stored in device data 198₁, 198 _(N), the personal information service 204 can determine wherethe user is located and where other devices of users matching the user'scriteria are positioned within the location.

The local copies of the user profile data 197 ₁, 197 _(N) may store someof the same user profile data 197 and may periodically update theirlocal copies with the user profile data stored by the computer system 12in an accessible database 197 over a communication network 50. Someexamples of user profile data 197 are the user's expressed preferences,the user's friends' list or more generally identification data for otherpeople having a stored relationship link to the user, the user's resume,the user's profession, family members, contacts, the user's preferredactivities, the user's favorites, some examples of which are, favoritecolor, favorite foods, favorite books, favorite author, etc., a list ofthe user's reminders, the user's social groups, the user's currentlocation, and other user created content, such as the user's photos,images, field of view image data captured by the front facing cameras,recorded videos and other user-specific information. In one embodiment,user profile data may include data obtained from one or more datasources or applications such as a user's social networking sites,contacts or address book, schedule data from a calendar application,email data, instant messaging data, or other sources on the Internet aswell as data directly entered by the user. Trust levels may bedetermined by user profile data 197 which identifies people known to theuser, for example as social networking friends and family memberssharing the same gaming service, or as employees in the same officewhich may be subdivided into different groups based on trust levels.Additionally, the user may explicitly identify trust levels in theiruser profile data 197 using the client side PI service application 204_(N). In one embodiment, the cloud based personal information serviceengine 204 aggregates data from user profile data 197 _(N) stored on thedifferent user computer systems 8, 161 of the user.

Each version of the personal information service application 204 alsostores in user profile data 197 a tracking history of the user. Someexamples of events, people and things tracked in the tracking historyare locations visited, transactions, content and real things purchased,field of view history, and people detected with whom the user hasinteracted. If electronically identified friends (e.g. social networkingfriends) are registered with the personal information service engine 204too, or they make information available to the user or publicly throughother applications 166, the personal information service engine 204 canuse this data as well to track the content and context of the user.

As previously mentioned, instead of a cloud based personal informationservice engine, the engine can 204 can operate on a server dedicated fora certain entity or location. A user can register for an event at alocation, such as a conference center, and create a user profile withinformation he wants made available for the event only. Differentauthorization settings 205 may be set identifying who may see theinformation. For example, permission or authorization settingscategories may be conference specific ones like employers or vendors.They may also be more general ones like those on my friends list orLinkedIn® contacts.

Also illustrated in this embodiment of the computing environment 54, oneor more person selection criteria 201 may have been received and storedbased on user input and one or more personal identification data sets202 associated with people who satisfy the personal selection criteriamay, at least temporarily, be stored in memory. Person selectioncriteria 201 generally relate to one or more characteristics,relationships, interests or experiences of a person, and user profiledata is searched for data indicating a match, within a criteria such asthat a commonly accessible search engine (Google®, Yahoo®) may use, withthese characteristics, relationships, interests and experiences issearched to identify a person satisfying the criteria. A characteristiccan be a physical characteristic such as height and gender.Characteristics may also be non-physical in nature such as social,political, religious, economic and employment related characteristicslike an occupation. User profile data can identify relationships withother people, organizations, and things both concrete e.g. carownership, and abstract. Some examples of interest are hobbies andfields of knowledge such as an interest in the latest discoveries aboutthe universe. Experiences can cover a wide range of life experiencesfrom the practical such as work related experience to fun ones such astravel.

In some examples, the personal information service engine 204 identifiespeople in the location having a relationship link in the user profiledata of the user, for example people who are on the user's friends listor contacts folder, and is able to search their user profile data fordata satisfying the person selection criteria in accordance with theaccess the related people, e.g. the friends, have granted the user. Forexample, if the user named Bob is in a first friend category for aperson named Pete who allows access to his resume only for those in thefirst friend category, the engine 204 can search the resume for datasatisfying the person selection criteria of Bob. However, the engine 204would not provide data from Pete's resume as data satisfying the personselection criteria of either Sue, who is in a second, less restrictivefriend category, or Lee, who would qualify as being in a category whichthe general public can access.

Pete has not allowed access to his resume Sue who is in a second friendcategory which searching user profile data of one or more persons havinga relationship link in the user profile data of the user for data whichsatisfies the person selection criteria and for which the user has beenpermitted or authorized access

Some examples of data fields which may be included in a personalidentification data set 202 are a personal identifier like a name, apersonal information data item which satisfies the person selectioncriteria, and a position with respect to the user. An example of apersonal information data item is a section from a resume about one'sJava experience which corresponds to person selection criteria for fiveyears of experience programming in Java. A personal identifier may alsoinclude an object identifier associated with the person from image dataof the location. Position data may also include a relative position interms of a direction and a distance range of the person from the user.

User authorization settings 205 identify what the user has grantedpermission for the personal information service engine 204 to share. Forexample, a participant may allow certain personal information items tobe shared at work or with certain people. Authorizations can also be setbased on location. A user may set that he or she does not wish to sharehis or her field of view data or his or her name as a personalidentifier in identity data sets. The PI service engine 204 can use analphanumeric code as an object identifier for the person instead. Thecode may also be encrypted for further privacy. In this way, the PIservice engine 204 and the location mapping engine 203 can track theuser's location with data received from devices in the location whilenot sharing identity data with such devices, while not sharing identitydata with other devices in the location.

Embodiments of methods for the technology and example implementationprocesses for some of the steps of the methods are presented in figuresbelow. For illustrative purposes, the method embodiments below aredescribed in the context of the system embodiments described above.However, the method embodiments are not limited to operating in thesystem embodiments described above and may be implemented in othersystem embodiments.

FIG. 4 is a flowchart of an embodiment of a method of operation of amixed reality display device system 8 which provides contextual personalinformation. In step 302, the display device system 8 identifies realobjects in a field of view of its see-through display device, and instep 304, which virtual data to display in the field of view by the oneor more image generation units 120 is determined based on executingapplications including a client side personal information (PI) serviceapplication 204 ₁. In step 306, the processing unit 210 under control ofthe client side PI service application 204 ₁ outputs data whichidentifies one or more people in the field of view who satisfy personselection criteria. In some examples, the outputted data may be a visualindicator registered to the one or more people As part of contributingto the 3D image mapping of the location, in step 308, the display devicesystem 8 transfers sensor data including field of view image data to thelocation mapping engine 203, and optionally, in step 310, transfersnon-image position and identity data received for surroundingcommunicatively coupled devices in accordance with user authorizationsettings for the surrounding devices. The non-image position data isdirectional position data when received by directional sensors 145 ofthe display device system 8. The steps of FIG. 4 are continuouslyperformed as the display device system 8 is operating and is executingthe PI service engine 204 ₁.

Similarly, the processing examples of FIGS. 5 through 8 are continuouslyperformed resulting in updates of personal identification data sets andnew generation of new data sets as people leave and enter the locationor as criteria changes based on user input.

In some instances, the user may select a focus identification modeinstead of a field of view identification mode for example as user inputduring the initialization of the client side application 204 ₁ or viavoice or gesture commands during execution of the application 204 ₁. Ina focus identification mode, data which identifies a person satisfyingthe personal information selection criteria is displayed when such aperson is an object of focus rather than simply being in the field ofview. A user may select this mode to avoid visual clutter. An object offocus may be identified as the point of gaze which is determined by theeye tracking software 196 based on image data of the eyes received fromthe eye tracking assemblies 134. In a field of view identification mode,any person satisfying the person selection criteria within the field ofview of the user's head mounted, mixed reality, display device systemhas identifying data such as a visual indicator registered to theperson, even if another person or object is the user object of focus.

FIG. 5 is a flowchart of an embodiment of a method for providingcontextual personal information to a mixed reality display devicesystem. For illustrative purposes, the embodiment of FIG. 5 is discussedin the context of being performed by a cloud based personal informationservice engine or application 204. In step 352, a request for dataidentifying any person sharing the location and satisfying personselection criteria is received from a user head mounted, mixed realitydisplay device system, and the request indicates a user location and theperson selection criteria. The user location data may be GPS dataidentifying a location, for example. It may also be image data of theuser in an image mapped location for which location data is known. Someexamples of a location may be a school, a company building, a hotel, aconference center or an outdoor recreational facility. In step 354, itis determined whether there is any person identified as sharing thelocation and satisfying the person selection criteria based onaccessible user profile data, e.g. the aggregated user profile data 197of users with accounts to the PI service engine 204.

For example, a current GPS location for people in the location may bestored in their respective user profile data accessible by the PIservice engine 204, and this location data can be compared with the GPSdata of the user to determine whether they share the same location. Inother examples, image data of people in the location captured by otherdisplay device systems 8 or positional cameras 52 in the location may beused to identify people based on facial recognition compared with imagedata stored in or accessible by their user profile data. In the case ofa person having a relationship link defined and stored in the userprofile data, e.g. a friend, if GPS data is not available or notavailable at a refined enough level, the PI service engine 204 cansearch the person's user profile data and identify the person's locationfrom data stored in the person's user profile data like a post onFoursquare® or Facebook® or a tweet on Twitter® indicating her currentlocation. As described above, the user profile data of the people in thelocation registered with the PI service engine 204 and publicinformation accessible for anyone identified in the location can besearched for matches within a criteria with the person selectioncriteria.

There may be people in the location who satisfy the person selectioncriteria, but if their privacy or user authorization settings 205 do notallow their data to be accessed or shared, such people cannot beidentified in the location.

If such a person is not identified or user authorization settings forsuch a person do not allow him or her to be identified, at least for thepersonal information related to the person selection criteria, then instep 356, the cloud based PI service engine 204 outputs data to the userthat no person can currently be identified in the location.

In step 358, the PI engine 204 generates a respective personalidentification data set for each person identified as satisfying theperson selection criteria and sharing the location. An illustrative loopstructure is used to show the processing of determining which of thepeople identified are within the user field of view and which areoutside of the field of view and the processing based on the result ofthe determining step. In step 359, a loop counter, i, is initialized tobegin processing of the people identified as sharing the user locationand satisfying the person selection criteria. In step 360, the PIservice engine 204 determines whether the person(i) is currently withina field of view of the user mixed reality, display device system basedon received field of view data. In step 362, responsive to the person(i)not being currently within the field of view of the user display devicesystem, the PI service engine 204 determines a relative position(i) ofthe person(i) to the user within the location, and stores in step 363the relative position(i) in the respective personal identification dataset(i). In step 364, responsive to the person being currently within thefield of view of the user display device system, the PI service engine204 stores an object identifier(i) for the person(i) in the field ofview in the respective personal identification data set(i). Whether theperson(i) is in the field of view or not, the PI service engine 204sends in step 365 the respective personal identification data set(i) tothe user display device system. The counter, i, is incremented in step366, and if it does not exceed N representing the number of people inthe location who satisfy the person selection criteria, the loop repeatsat 359 for another identified person. When the counter, i, exceeds N,the loop ends at step 367.

FIG. 6 is a flowchart of an embodiment of a method for providingcontextual personal information by a head mounted, mixed reality displaydevice system. The client PI application 204 ₁ in step 402 receives andstores person selection criteria associated with a user. The user mayhave set up criteria which applies in any environment or location via aprevious online session which has been stored by the cloud based side204 of the PI engine. The display device system 8 downloads the personcriteria and stores locally. In other examples, person selectioncriteria is received locally, for example via input to the processingunit 4 or via physical action user input. A user may for example say,please identify all the single people at a concert event, and the clientside 204 ₁ of the PI application sends the criteria to the cloud basedside, and receives and displays identifying data for any personidentified who satisfies the criteria in the field of view.

In line with the example just mentioned, in step 404, the PI engine 204₁ sends a request indicating a user location and the person selectioncriteria for a respective personal identification data set for eachperson identified as sharing the location and satisfying the personselection criteria to a personal identification service engine like thecloud based personal identification service application or engine 204.In step 406, a respective personal identification data set for eachperson identified as sharing the location and satisfying the personselection criteria from the personal identification service engine isreceived.

An illustrative loop structure is used to show the processing ofdetermining which of the people identified are within the user field ofview and which are outside of the field of view and the processing basedon the result of the determining step. In step 407, a loop counter, i,is initialized to begin processing of the number of personalidentification data sets N which have been received.

In step 408, the PI application 204 ₁ interfaces with the objectrecognition software 192 for determining whether a person(i) associatedwith and identified by a personal identifier(i) in a respective personalidentification data set(i) is identified in the field of view of thedisplay device system. For example, facial recognition software of theobject recognition software 192 may be applied to person objects in thefield of view for identifying the person(i). Additionally, non-imagedirectional position data from surrounding devices may identify theperson(i) is in the field of view, for example the back of the head ofthe person(i) may be right in front of the user.

If the person(i) is not identified in the field of view of the user'sdevice, in step 413, the PI application 204 ₁ retrieves a position(i)within the location for the person(i) from the personal identificationdata set(i), and in step 414 causes the image generation unit 120 todisplay the personal identifier(i) and an indicator of the position(i)within the location for the person(i). Some examples of the indicator ofthe position(i) may be one or more directional symbols, directions intext or images, or a place identifier, and particularly the last one ifthe location is one the user tracking history in user profile data 197indicates the user has visited a number of times or for an extendedperiod of time.

Responsive to the person(i) associated with the respective personalidentification data set(i) being identified in the field of view, instep 410, the PI application 204 ₁ causes the image generation unit 120to output data which identifies the person(i) in the field of view. Someexamples of such data are a visual effect like highlighting, the text,image data, or video data, whether two dimensional or 3D included as apersonal information item in the personal identification data set(i). Inother examples, the data output which identifies the person(i) in thefield of view is self-selected image data included in the personalinformation item or another data item in the personal identificationdata set which the person(i) has designated to be displayed to identifyhim or her. An example of such self-selected image data is a hologramregistered to the person(i) in the field of view. An example of ahologram may be an avatar which overlays the person(i).

In a different embodiment with a device system which includes camerasbeside front facing cameras, image data from a non-front facing camerahaving person(i) recognized in its field of view may be displayed in themixed reality see-through display with visual data identifying theperson in the image data. Optionally, in step 412, the client side PIapplication 204 ₁ sends a message to the cloud based PI service engine204 with the personal identifier(i) stored in the data set(i) for theperson(i) indicating the person(i) is in the field of view.

The counter, i, is incremented in step 415, and if it does not exceed Nrepresenting the number of received personal identification data sets,the loop repeats at 407 for another identified person. When the counter,i, exceeds N, the loop ends at step 417.

FIGS. 7A and 7B illustrate examples of implementation processes whichmay be used for determining whether a person is identified within afield of view of the user and determining a position of the person ifnot identified in the field of view. The person is typically identifiedin image data via facial recognition or pattern recognition such ascomparison with a skeletal model or some combination of these. Inprocessing image data for motion tracking, often people and things inimage data are assigned object identifiers to track their motion. Insome examples, the cloud based PI engine 204 assigns a unique objectidentifier to each person in a location, and tracks a position of eachof the one or more users within the location based on an image mappingof the location. The unique object identifier can then be used by alldisplay devices in the location for tracking objects. For determiningwhether an object associated with the unique object identifier assignedto the person is within the field of view of the user display devicesystem, the PI engine 204 correlates image data received from at leastone field of view camera of the user display device system and theposition of the person in the image mapping of the location.

Some examples of a location can be a home, a work place, a campus, ageofenced area, a conference center, a room at a conference center andsmaller and bigger locations as a user may indicate via user input datato the PI service engine or application 204, 204 ₁.

FIG. 7A is a flowchart of an embodiment of a process for determiningwhether a person associated with a respective personal identificationdata set can currently be identified within a field of view of theuser's mixed reality display device system. The process is discussedfrom the perspective of the display device system 8, but may also beperformed by a PI service engine 204 executing remotely with access tothe user field of view data.

In step 452, the client PI service application 204 ₁ determines whetherfacial recognition has identified an object in the field of view as theperson(i). If so, in step 454, the application 204 ₁ returns to executestep 410. If facial recognition has not identified the person(i), theclient PI service application 204 ₁ in step 456 tracks identity datasets including a personal identifier and a sensor head position receivedat each sensor at a head position of the user display device system fromone or more head mounted devices within range of each of the headpositioned sensors of the user display device system. In otherembodiments, the sensor position may be referenced to another body partother than the head or be a particular position on the head.

In step 458, the application 204 ₁ determines whether any of thepersonal identifiers indicate the person(i) associated with therespective personal identification data set(i). If not, the client PIservice application 204 ₁ returns to step 413 in step 460. If theperson(i) is associated with one of the personal identifiers, then instep 462, based on head position data of the sensors of the user displaydevice and the within range head mounted, mixed reality, display devicesystems, the client PI service application 204 ₁ determines aposition(i) of the person(i) with respect to the user position. In step464, based on the determined position(i), it is determined whether theperson(i) is identified in the field of view. For example, the objectrecognition engine 192 can provide identifying information like personalor physical features height, body shape, skin color, hair color, eyecolor, etc. for all person objects in the field of view to the client PIservice application 204 ₁. With the direction and distance of adetermined position(i) for a person and personal identification datafrom the user profile data of the person(i) like hair color, height andskin color, the client PI service application 204 ₁ may execute apredetermined weighted average algorithm based on the position andidentified personal features to determine whether a person object in thefield of view is the person(i) based on a probabilistic criteria.

If the person(i) is not identified in the field of view, in step 466,the client PI service application 204 ₁ returns to step 414, but if theperson(i) is identified in the field of view, the application in step468 returns to step 410 and causes the display device system 8 to outputidentifying data for the person(i) in the field of view.

The sound recognition engine 194 can also aid in identifying theperson(i) by indicating whether a voice identified with the person basedon voice recognition data in the person(i)'s user profile data is beingdetected, and can identify the voice for the user as that of theperson(i). The user can then listen for the voice around him and movehis head in the direction of the voice if heard which will aid theapplication 204 ₁ in visually identifying the person(i).

FIGS. 7B, 7C and 7D are flowchart of an embodiment of a process fordetermining a position of a person associated with a respective personalidentification data set within the location responsive to the person notidentified as being within the user field of view. FIG. 7B is aflowchart of an embodiment of a process for determining a position of aperson associated with a respective personal identification data set andnot identified within the user field of view but who is associated witha head mounted, mixed reality display device system in the location.FIG. 7B is discussed for illustrative purposes from the perspective ofthe cloud based PI service engine 204 which can have access to data fromdisplay devices and other computing devices throughout the location. Instep 502, the cloud based PI service engine 204 determines whether therehas been identified a head mounted, mixed reality display device systemin the location associated with the personal identifier(i) of theperson(i) associated with the respective personal identification set(i)capturing field of view data. Responsive to such a device beingidentified, in step 506, the PI engine 204 retrieves field of view imagedata from the display device system associated with the personalidentifier(i) of the person(i) , and based on image mapping data of thelocation and the field of view image data in step 510, identifies aposition(i) of the person(i) within the location.

In step 512, the PI engine 204 identifies a position of the user headmounted, mixed reality, display device system in the location, and instep 514, generates virtual data for directing the user to the person(i)in the location which is sent in step 516 to the user display devicesystem.

If the determination in step 502 had determined that a head mounted,mixed reality display device system associated with the person(i)capturing field of view data has not been identified, the cloud based PIservice engine 204 proceeds to step 504 of FIG. 7C.

FIG. 7C is a flowchart of an embodiment of a process for determining aposition of a person within the location and associated with arespective personal identification data set responsive to the person notbeing within the user field of view but who has been identified in fieldof view data of another head mounted, mixed reality, display devicesystem.

In step 504, the PI service engine 204 determines whether there is ahead mounted, mixed reality, display device system associated with asecond personal identifier in the location identifying the person(i)associated with the at least one personal identification data set(i) isin its field of view. The second personal identifier identifies a wearerof an HMD other than the person(i) or the user. In other words, if theuser does not have the person in her field of view, and the person isnot wearing a device or transmitting field of view data, someone elsemay be uploading field of view data which facial recognition softwarehas processed and has identified the person(i). If there is such adisplay device system associated with a second personal identifier inthe location which has the person(i) identified in its field of viewdata, the cloud based PI engine 204 in step 526 retrieves field of viewimage data from the display device system associated with the secondpersonal identifier, and in step 528 identifies a position(i) of theperson(i) within the location, based on image mapping data of thelocation and the field of view data retrieved from the display devicesystem associated with the second personal identifier. Steps 512, 514and 516 are then performed based on the identified position(i) and theuser position.

FIG. 7D is a flowchart of an embodiment of a process for determining aposition of a person within the location and associated with arespective personal identification data set responsive to the person notbeing within the user field of view but for whom non-image location dataindicates a position.

If there is not a device associated with a second personal identifierwhich has the person(i) identified in its field of view, then the PIservice engine 204, determines in step 508 whether there is non-imagelocation data identifying a relative position (i) of the person(i)within the location from a head mounted, mixed reality display devicesystem associated with a second personal identifier. As mentioned above,non-image location data can be based on directional sensor data from oneor more sensors, each having a known range. The non-image location datacan be from one or more devices surrounding the person(i) associatedwith the respective personal identification data set(i).

If there is non-image location data indicating a position of the personwithin the location from a head mounted, mixed reality, display devicesystem associated with a second personal identifier, the PI engine 204retrieves field of view image data from the display device systemassociated with the second personal identifier in step 536. In step 538,based on image mapping data of the location and the field of view imagedat, a position within the location of the display device systemassociated with the second identifier is identified. In step 540, aposition(i) of the person(i) within the location is identified based onthe relative position(i) and the identified position of the displaydevice system associated with the second personal identifier. Theprocess continues by performing steps 512, 514 and 516.

If there is no non-image location data from which a relative position(i)within the location to the user can be determined, then the PI engine204 in step 518, outputs data to the user that the person(i) has beenidentified as being within the location or the most detailed positiondata the engine 204 currently has for the person(i).

In some examples, the non-image data can also come from the deviceassociated with the person sought in the form of an identity set whichis transferred to a directional sensor in the location.

The PI service engine 204 also can track personal interactions or lackof interactions with other users for the user. For example, if a user ata software gaming conference sets person selection criteria of peoplewith a virtual machine language for gaming applications Zebron (afictitious name used only for illustrative purposes), the PI serviceidentifies the people for the user either in the user field of view orwith position data. The service can also track whether the user paidattention to the identified people or not while at the conference. Apersonal interaction can be identified for each identified person whichmay indicate an attention level. Personal interactions can also betracked for people not part of the set of people satisfying personselection criteria.

FIG. 8A is a flowchart of an embodiment of an overall process fortracking personal interactions with one or more people in a location. Instep 532, the PI service engine or application 204, 204 ₁ determines apersonal interaction status for a person in the field of view based onaudio and image data, and in step 534, stores and outputs a personalinteraction status for the person.

FIG. 8B is a flowchart of an embodiment of a process for determining apersonal interaction status for a person. In 552, the PI serviceapplication 204, 204 ₁ identifies one or more physical actions of aperson in the field of view of the user's display device determined tobe directed at the user.

For example, the person may be waving but his head position, and eyeposition if captured, indicates he is not waving at the user. Aninteraction status for such a person would be none.

In step 554, the PI service application 204, 204 ₁ identifies one ormore user physical actions while the person is an object of focus andwithin a time period of the one or more physical actions of the persondetermined to be directed at the user. The time period may be before orafter the one or more physical actions of the person. For example, eyedata indicates the person is an object of focus, the user waved her handin the field of view of her display device system 8, and the person madea hand wave while his face is centered in her field of view image datafrom cameras 113. Such an interaction would illustrate a gestureacknowledgement type of personal interaction.

In step 556, the PI service application 204, 204 ₁ identifies one ormore personal interaction status candidates which may apply based on theone or more physical actions of the person and the user, and in step 558selects as a personal interaction status the candidate indicating thehighest level of interaction. For example, the PI service application204, 204 ₁ accesses and implements the personal interaction rule set 178for linking one or more physical actions and data indicating at whom theone or more physical actions are directed.

Some examples of personal interaction statuses in order of increasinglevel of interaction are none, a gesture acknowledgement, a spokengreeting, conversation, conversation related to user profile data, andconversation related to person selection criteria. A spoken greeting maybe detected from audio data received via the microphone 110 based on adictionary in the personal interaction rule set 178 of customary wordsindicating greeting in the one or more languages used in the location.For example, the user saying “hello” while gaze data indicates a personin her field of view, and her field of view image data indicating aperson looking at her within a time period with lips moving and thewords “hi” being received by the microphone of a voice not the user'swould indicate a spoken greeting.

A continuation of spoken words while the person appears to be focused onthe user in her field of view image data, and her gaze data indicatesthe person continues to be an object of focus for the user indicatesconversation. Sound recognition software 194 can compare the voice ofthe person to voice profiles to see if the person is a known associateof the user based on user profile data if the person has not beenidentified already. The sound recognition software 194 can convert theaudio data to be put into a text format which the PI service application204, 204 ₁ can search for identifying words spoken related to userprofile data and the person selection criteria. If so, the personalinteraction status can be one of conversation related to user profiledata, conversation related to person selection criteria or both.Keywords associated with the aspect of user profile data or personselection criteria discussed can be selected and stored in memory aswell. If the user has a personal identification data set stored for theperson, the personal interaction status and any related keywords can bestored in the user's copy of the personal identification data set forthe person.

Such interaction characterizations can help jog the user's memory aboutthe different people she has met. They can also indicate hereffectiveness at making connections with intended people. For example,if she had conversation related to person selection criteria of peopleworking on HMD devices with two out of ten people she did not know, butidentified for her at a mixed reality professional conference, but spentan hour talking about skydiving with her office mate at the conference,she needs to improve her work related knowledge sharing efforts.

FIG. 9 illustrates an example of providing contextual personalinformation to a user wearing a head mounted, mixed reality displaydevice system. In this example, a user, for example Betty, is attendingthe annual “Zebron” conference which many colleagues and people in herfield of game development attend. The personal information serviceengine 204 has determined Betty's location based on GPS data as aparticular conference center, and has no data indicating she has beenthere before.

The PI engine 204 has identified a social context as work and asub-setting of conference and has determined the subject matter of theconference is “Zebron” a popular virtual machine language used by gamedevelopers. Betty enters person selection criteria of experience withZebron 2 or Zebron 1. Additionally, Betty enters another personselection criteria for any person who shares common aspects of hereducational and work experience, for example those with the same major,same college, same company or same languages programmed

Fortunately, other people at the conference have also registered withthe PI service engine 204 or grant temporary permission for access totheir user profile data from one or more applications for the time andplace of the conference. For example, Bob Hitosh has allowed informationfrom his online resume relating to Zebron to be made available forothers with mixed reality display device systems 8 for the conference.Alexi Gusparov has permitted the PI service 204 to release informationfrom his resume generally that may have a relation to others at theconference. Public information on the conference's website has providedimage data for Dr Kim and that he will be presenting on the topic of“Optimizations for Zebron 2.”

The PI service engine 204 sends personal identification data setsincluding the person selection criteria they relate to for Dr. Kim, SueThompson and Bob Hitosh to Betty for the Zebron related person selectioncriteria. The PI service engine 204 also sends a personal identificationdata set for Alexi Gusparov to Betty's device system as Alexi went toCarnegie Mellon University like Betty and majored in computer science.

The client PI application 204 ₁ identifies Dr Kim 28 in Betty's field ofview, and displays text on the wall 54 behind him as he considers aquestion from Bob. The text includes a description from the conferencewebsite of “Dr. Larry Kim Presenter: “Optimizations for Zebron 2”Identity data sets may also have been exchanged between directionalsensors of the display device systems 8 of Dr. Kim, shown as a displaydevice 2 and a mobile device as a processing unit 4, and Betty's systemwhich uses a wrist based processing unit 4.

The cloud based PI service engine 204 has identified Alexi is in theroom via GPS, and Betty's display device system 8 has exchanged identitydata sets with Alexi's smartphone 4 even though Alexi is not wearing hisdisplay device 2 like Dr. Kim 28 and Bob 29. Based on the position ofAlexi in the room, the client PI application 204 ₁ identifies Alexi inBetty's field of view and displays on his shirt, his name and “CMU 03.”

Betty knows Bob 29 outside of work, but the cloud based PI engine 204identifies an accomplishment of Bob's for Zebron from his resume itemshe has permitted to be shared as the PI service engine 204, 204 ₁ has norecord of Betty viewing Bob's online resume. Bob has been recognizedalso by directional sensors on Betty's display device 2 picking upidentity data sets emitted from sensors 145 on the back of the templesof Bob's display device 2. Text from Bob's resume is displayed on Bob'sshirt: “Bob Hitosh Developer of “Antwar” in Zebron beta version 1.”

In other examples, each of Bob, Dr. Kim, Alexi and Betty may also havebeen tracked by location sensors, for example cameras, in the conferencecenter and their positions relayed by the PI service engine 204 toBetty's client PI application 204 ₁.

Sue Thompson is not in Betty's field of view. The PI service engine 204has identified Sue's position and sends virtual data for directions toBetty's device system 8 which the client side PI application 204 ₁causes to display. The virtual data is a series of arrows which directBetty to Sue from Betty's current location. In the illustrated example,a virtual arrow 50 stating “Sue Thompson” appears integrated as apainted arrow on the wall 54 pointing to the right of Auditorium A. Thearrows will update as Betty or Sue moves, and Betty can clear thedisplay if she wishes as well via a user input command.

FIG. 10 is a block diagram of one embodiment of a computing system thatcan be used to implement one or more network accessible computingsystems 12 which may host at least some of the software components ofcomputing environment 54 or other elements depicted in FIG. 3. Withreference to FIG. 16, an exemplary system for implementing the inventionincludes a computing device, such as computing device 800. In its mostbasic configuration, computing device 800 typically includes one or moreprocessing units 802 and may include different types of processors aswell such as central processing units (CPU) and graphics processingunits (GPU). Computing device 800 also includes memory 804. Depending onthe exact configuration and type of computing device, memory 804 mayinclude volatile memory 805 (such as RAM), non-volatile memory 807 (suchas ROM, flash memory, etc.) or some combination of the two. This mostbasic configuration is illustrated in FIG. 10 by dashed line 806.Additionally, device 800 may also have additionalfeatures/functionality. For example, device 800 may also includeadditional storage (removable and/or non-removable) including, but notlimited to, magnetic or optical disks or tape. Such additional storageis illustrated in FIG. 10 by removable storage 808 and non-removablestorage 810.

Device 800 may also contain communications connection(s) 812 such as oneor more network interfaces and transceivers that allow the device tocommunicate with other devices. Device 800 may also have input device(s)814 such as keyboard, mouse, pen, voice input device, touch inputdevice, etc. Output device(s) 816 such as a display, speakers, printer,etc. may also be included. All these devices are well known in the artand need not be discussed at length here.

As discussed above, the processing unit 4 may be embodied in a mobiledevice. FIG. 11 is a block diagram of an exemplary mobile device 900which may operate in embodiments of the technology. Exemplary electroniccircuitry of a typical mobile phone is depicted. The phone 900 includesone or more microprocessors 912, and memory 910 (e.g., non-volatilememory such as ROM and volatile memory such as RAM) which storesprocessor-readable code which is executed by one or more processors ofthe control processor 912 to implement the functionality describedherein.

Mobile device 900 may include, for example, processors 912, memory 1010including applications and non-volatile storage. The processor 912 canimplement communications, as well as any number of applications,including the applications discussed herein. Memory 1010 can be anyvariety of memory storage media types, including non-volatile andvolatile memory. A device operating system handles the differentoperations of the mobile device 900 and may contain user interfaces foroperations, such as placing and receiving phone calls, text messaging,checking voicemail, and the like. The applications 930 can be anyassortment of programs, such as a camera application for photos and/orvideos, an address book, a calendar application, a media player, aninternet browser, games, other multimedia applications, an alarmapplication, other third party applications like a skin application andimage processing software for processing image data to and from thedisplay device 2 discussed herein, and the like. The non-volatilestorage component 940 in memory 910 contains data such as web caches,music, photos, contact data, scheduling data, and other files.

The processor 912 also communicates with RF transmit/receive circuitry906 which in turn is coupled to an antenna 902, with an infraredtransmitted/receiver 908, with any additional communication channels 960like Wi-Fi, WUSB, RFID, infrared or Bluetooth, and with amovement/orientation sensor 914 such as an accelerometer. Accelerometershave been incorporated into mobile devices to enable such applicationsas intelligent user interfaces that let users input commands throughgestures, indoor GPS functionality which calculates the movement anddirection of the device after contact is broken with a GPS satellite,and to detect the orientation of the device and automatically change thedisplay from portrait to landscape when the phone is rotated. Anaccelerometer can be provided, e.g., by a micro-electromechanical system(MEMS) which is a tiny mechanical device (of micrometer dimensions)built onto a semiconductor chip. Acceleration direction, as well asorientation, vibration and shock can be sensed. The processor 912further communicates with a ringer/vibrator 916, a user interfacekeypad/screen, biometric sensor system 918, a speaker 920, a microphone922, a camera 924, a light sensor 921 and a temperature sensor 927.

The processor 912 controls transmission and reception of wirelesssignals. During a transmission mode, the processor 912 provides a voicesignal from microphone 922, or other data signal, to the RFtransmit/receive circuitry 906. The transmit/receive circuitry 906transmits the signal to a remote station (e.g., a fixed station,operator, other cellular phones, etc.) for communication through theantenna 902. The ringer/vibrator 916 is used to signal an incoming call,text message, calendar reminder, alarm clock reminder, or othernotification to the user. During a receiving mode, the transmit/receivecircuitry 906 receives a voice or other data signal from a remotestation through the antenna 902. A received voice signal is provided tothe speaker 920 while other received data signals are also processedappropriately.

Additionally, a physical connector 988 can be used to connect the mobiledevice 900 to an external power source, such as an AC adapter or powereddocking station. The physical connector 988 can also be used as a dataconnection to a computing device. The data connection allows foroperations such as synchronizing mobile device data with the computingdata on another device.

A GPS receiver 965 utilizing satellite-based radio navigation to relaythe position of the user applications is enabled for such service.

The example computer systems illustrated in the figures include examplesof computer readable storage devices. A computer readable storage deviceis also a processor readable storage device. Such devices may includevolatile and nonvolatile, removable and non-removable memory devicesimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules orother data. Some examples of processor or computer readable storagedevices are RAM, ROM, EEPROM, cache, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other optical diskstorage, memory sticks or cards, magnetic cassettes, magnetic tape, amedia drive, a hard disk, magnetic disk storage or other magneticstorage devices, or any other device which can be used to store thedesired information and which can be accessed by a computer.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. One or more processor-readable storage deviceshaving instructions encoded thereon for causing one or more softwarecontrolled processors to execute a method for providinglocation-relevant contextual personal information by a mixed realitydisplay device system, the method comprising: receiving and storingperson selection criteria having been provided by a user wearing a mixedreality display device of the system, the person selection criteriabeing for identifying another person who satisfies the person selectioncriteria; sending a request including a location of the user and theperson selection criteria to a personal information service engineexecuting on one or more remote computer systems for a personalidentification data set for each person sharing the location andsatisfying the person selection criteria, the location being one sharedby the user and one or more other persons such that face to facemeetings can occur at the location between the user and the one or moreother persons, the location including a scene in a field of view of themixed reality display device; receiving at least one personalidentification data set from the personal information service engine fora person sharing the location; determining whether the person associatedwith the at least one personal identification data set is in the fieldof view of the mixed reality display device; responsive to the personassociated with the at least one personal identification data set notbeing currently within the field of view of the mixed reality displaydevice, determining a position of the person within the location, andoutputting data which indicates the out-of-field-of-view position of theperson within the location; and responsive to the person associated withthe at least one personal identification data set being in the field ofview, outputting data which identifies the in-field-of-view position andidentity of the person in the field of view.
 2. The one or moreprocessor-readable storage devices of claim 1 further comprising:determining a ready-for-face-to-face personal interaction status for anin-field-of-view person associated with the at least one personalidentification data set based on audio and image data captured by themixed reality display device system; and outputting a personalinteraction status stored for the person.
 3. The one or moreprocessor-readable storage devices of claim 2 wherein theready-for-face-to-face personal interaction status identifies one ormore of the following: none; that interaction status is that associatedwith a greeting; that interaction status is that associated with aconversation; that interaction status is that associated with aconversation related to personal information selection criteria; thatinteraction status is that associated with a conversation related touser profile data; and that interaction status is that associated with agesture acknowledgement.
 4. The one or more processor-readable storagedevices of claim 2 wherein the determining of the ready-for-face-to-facepersonal interaction status for the in-field-of-view person associatedwith the at least one personal identification data set based on audioand image data captured by the mixed reality display device systemfurther comprises: identifying one or more interactions from thein-field-of-view person within a time period of the person coming intothe field of view of the mixed reality display device; identifying oneor more user physical actions within the time period of the one or moreinteractions while the in-field-of-view person is an object of interest;identifying one or more personal interaction status candidates which mayapply based on the one or more interactions and the one or more userphysical actions within the time period; and selecting as a personalinteraction status the candidate indicating the highest level ofready-for-face-to-face personal interaction.
 5. The one or moreprocessor-readable storage devices of claim 1 wherein outputting datawhich identifies the in-field-of-view position and identity of theperson in the field of view further comprises: displaying anidentification providing visual indicator registered to thein-field-of-view position of the person in a display of the mixedreality display device.
 6. The one or more processor-readable storagedevices of claim 1 wherein outputting data which identifies thein-field-of-view position and identity of the person in the field ofview further comprises: playing audio including a personal informationitem of the identified in-field-of-view person which satisfies theperson selection criteria through at least one earphone of the mixedreality, display device system.
 7. The one or more processor-readablestorage devices of claim 1 further comprising: responsive to the personassociated with the at least one personal identification data set notbeing in the field of view, displaying a personal identifier and aposition within the location of the person associated with the at leastone personal identification data set.
 8. A method for providinglocation-relevant contextual personal information to a head mounted,mixed reality, display device of a mixed reality system, the methodcomprising: receiving a request indicating a location of a user of ahead mounted, mixed reality display device of the system and indicatingperson selection criteria for at least one personal identification dataset for each of other persons sharing the location with the user andsatisfying the person selection criteria, the shared location being suchthat face to face meetings can occur at the location between the userand one or more of the other persons at the location, the locationincluding a scene in a field of view of the mixed reality display deviceof the user; determining whether there is a person sharing the locationand satisfying the indicated person selection criteria based onaccessible user profile data; responsive to there being a person sharingthe location and satisfying the indicated person selection criteria,providing at least one personal identification data set for the person;determining whether the person associated with the provided at least onepersonal identification data set is currently within the field of viewof the head mounted, mixed reality, display device of the user;responsive to the person not being currently within the field of view ofthe display device determining a position of the person within thelocation, and sending an indication of the position in the at least onepersonal identification data set; and responsive to the person beingcurrently within the field of view of the display device of the user,sending the at least one personal identification data set including apositional indicator and a personal identifier for the person in thefield of view.
 9. The method of claim 8 wherein determining whetherthere is a person sharing the location and satisfying the personselection criteria based on accessible user profile data furthercomprises: searching user profile data of one or more persons having arelationship link in the user profile data of the user for data whichsatisfies the person selection criteria and for which the user has beenpermitted access.
 10. The method of claim 8 wherein the personalidentifier is an object identifier unique for the location.
 11. Themethod of claim 8 wherein determining whether the person associated withthe at least one personal identification data set is currently within afield of view of the user head mounted, mixed reality, display devicesystem further comprises: receiving image data of one or more users inthe location; assigning a unique object identifier to each of the one ormore users; tracking a position of each of the one or more users withinthe location based on an image mapping of the location; and determiningwhether an object associated with the unique object identifier assignedto the person is within the field of view of the user display devicesystem based on correlating image data received from at least one fieldof view camera of the user display device system and the position of theperson in the image mapping of the location.
 12. The method of claim 8wherein determining a position of the person within the location furthercomprises: determining whether there is a head mounted, mixed reality,display device system associated with the personal identifier of theperson in the location; and responsive to there being a display devicesystem associated with the personal identifier of the person in thelocation, retrieving from the display device system associated with thepersonal identifier field of view image data, identifying the positionof the person within the location based on image mapping data of thelocation and the field of view image data of the display device systemassociated with the personal identifier, identifying a position of theuser head mounted, mixed reality, display device system, generatingvirtual data for directing the user to the person in the location, andsending the virtual data to the user display device system.
 13. Themethod of claim 12 further comprising: responsive to there not being ahead mounted, mixed reality, display device system associated with thepersonal identifier of the person in the location, determining whetherthere is a second head mounted, mixed reality, display device systemassociated with another personal identifier in the location having theperson in its field of view; and responsive to there being a second headmounted, mixed reality display device system having the person in itsfield of view, retrieving field of view image data from the seconddisplay device system, identifying the position of the person within thelocation based on image mapping data of the location and the field ofview image data from the second display device system, identifying aposition of the user display device system, generating virtual data fordirecting the user to the person in the location, and sending thevirtual data to the user display device system.
 14. The method of claim13 further comprising: responsive to there not being a head mounted,mixed reality, display device system associated with the personalidentifier of the person in the location, determining whether there is asecond head mounted, mixed reality, display device system associatedwith another personal identifier in the location having non-imagelocation data indicating a relative position of the person from aposition of the second display device system; and responsive to therebeing a second head mounted, mixed reality, display device system havingthe non-image location data indicating the relative position of theperson from the second display device system, identifying the positionof the second display device system within the location based on imagemapping data of the location and field of view image data of the seconddisplay device system, identifying the position of the person within thelocation based on the relative position of the person from the seconddisplay device system and the position of the second display devicesystem within the location, identifying a position of the user displaydevice system, generating virtual data for directing the user to theperson in the location, and sending the virtual data to the user displaydevice system.
 15. The method of claim 14 wherein the non-image locationdata is data from one or more directional sensors on the second displaydevice system.
 16. A head mounted, mixed reality display device systemfor providing contextual personal information comprising: a mixedreality display positioned by a head mounted support structure worn by auser; at least one front facing camera positioned on the supportstructure for capturing image data of a field of view of the mixedreality display; one or more directional sensors attached to the supportstructure, each having a sensor position with reference to a body partof the user and transmitting an identity data set including the sensorposition; one or more software controlled processors communicativelycoupled to the at least one front facing camera for receiving the imagedata of the field of view; the one or more software controlledprocessors being communicatively coupled to a remote computer systemexecuting a personal information service engine for sending a requestwith person selection criteria and a location of the user wearing thehead mounted support structure, for receiving a personal identificationdata set of one or more other persons sharing the location with the userand satisfying the person selection criteria, and for determiningwhether the person associated with the personal identification data setis in the field of view of the mixed reality display, where the locationshared by the user and the one or more other persons is such that faceto face meetings can occur at the location between the user and the oneor more other persons and the location includes a scene in a field ofview of the mixed reality display of the user; and at least one imagegeneration unit communicatively coupled to the one or more softwarecontrolled processors and optically coupled to the mixed reality displayfor, responsive to the person associated with the personalidentification data set being in the field of view, tracking virtualdata to an in-field-of-view position of the person, and responsive tothe person associated with the personal identification data set notbeing currently within the field of view, displaying image data whichindicates a position of the person within the location but outside thefield of view.
 17. The system of claim 16 further comprising: amicrophone supported by the support structure for capturing audio dataand being communicatively coupled to the one or more software controlledprocessors for sending audio data; one or more eye tracking assembliespositioned by the support structure for capturing image data of each eyeand communicatively coupled to the one or more software controlledprocessors for sending image data of each eye; the one or more softwarecontrolled processors identifying a person object of interest based onthe image data of each eye and image data of the field of view of themixed reality display; and the one or more software controlledprocessors determining a personal interaction status for the personobject of interest based on audio and image data captured of the personobject of interest and a personal interaction rule set stored in anaccessible memory.
 18. The system of claim 16 further comprising: amicrophone supported by the support structure for capturing audio dataand being communicatively coupled to the one or more software controlledprocessors for sending audio data; the one or more software controlledprocessors identifying a person object of interest based on a gesture ofthe user recognized in image data of the field of view of the mixedreality display; and the one or more software controlled processorsdetermining a personal interaction status for the person object ofinterest based on audio and image data captured of the person object ofinterest and a personal interaction rule set stored in an accessiblememory.
 19. The system of claim 17 further comprising: the one or moresoftware controlled processors storing in an accessible memory apersonal interaction status for the person satisfying the personselection criteria; and the at least one image generation unitdisplaying a personal interaction status for the person satisfying theperson selection criteria responsive to user input.
 20. The system ofclaim 16 wherein the one or more directional sensors comprise at leastone of the following: a directional antenna; an infrared device; aGlobal Positioning System (GPS) device; a radio frequency device; anetwork access point; a cellular telecommunication based device; and awireless Universal Serial Bus device.